This invention relates to a network interface device having a buffer memory shared between a plurality of physical network ports.
A network interface device (NIC) provides an interface between a physical network and a data processing system, such as a server or other computer system. The network interface allows the data processing system to transmit and receive data packets over the physical network so as to permit communication with other data processing entities on the network. In order to ensure that data packets are not dropped, a NIC must be able to receive and retain each data packet received from the network at the maximum data rate at which the network operates. This is generally achieved by providing a buffer memory for each physical data port of the NIC into which received data packets can be streamed prior to routing and/or processing by the NIC. The buffer memory available to each data port of a NIC must be sufficiently large to avoid back-pressuring incoming data streams.
Conventional network interface devices provide a dedicated FIFO (first-in-first-out) memory buffer of fixed size for each physical port of the NIC. An example of this architecture is shown in FIG. 1. Network interface device 1 has three data ports 3, 4 and 5 which provide connections to networks 2 (these could be the same or different physical networks). Each data port has a dedicated FIFO memory buffer 6, 7 or 8, into which data packets received over the respective data port are streamed. The received data packets are pulled off the FIFO memory buffers by an arbitrator 9, or other entity of the network interface device, which enforces flow control and directs the packets to the appropriate processing entity of the network interface device.
Each memory buffer should be large enough so that when data packets of the maximum packet size are being received over the corresponding physical port at the maximum data rate, the memory will not become full whilst the routing and/or processing portions of the NIC are handling other memory buffers, or are otherwise not attending to the memory buffer. Other pressures act to limit the size of each memory buffer: the high speed memory required for the buffers is expensive and each additional byte of buffer memory increases the power requirements and heat generated by the NIC. These pressures are of particular concern in NICs intended for use in datacentres because such NICs are often densely packed into server racks. The size of the memory buffers provided in any given NIC is a trade-off between these competing factors.
The dedicated FIFO memory buffers of a conventional NIC each comprise a plurality of fixed-size buffers 10, which are generally designed with the intention to accommodate one packet per buffer. This simplifies the handling of received data packets and is straightforward to implement at high speed. However, the use of fixed size buffers is inefficient: if the buffers are small relative to the maximum packet size then large packets will require linking, which takes up additional memory; and if the buffers are large enough to accommodate the maximum packet size this results in unused memory when smaller data packets are received which do not fill a buffer.
A further problem with NICs having multiple physical ports is that the conventional memory buffer structure leads to poor utilisation of the total amount of buffer memory available at the NIC. In a real-world scenario it is unlikely that, from one moment to the next, all of the ports experience an equal throughput in data. The buffers corresponding to less active ports might therefore be largely empty while the buffers of more active ports become full. Because a port cannot utilise the buffers of another port, much of the expensive buffer memory provided at a NIC can go unused.
Despite these limitations, the provision of dedicated memory buffers at a NIC to handle the incoming streams of data packets from each port has become the norm. This is largely a result of the following three factors:
I/O bus performance between a NIC and its host system has been outstripping network performance over the last few years in terms of bandwidth and jitter, which has meant that relatively small memory buffers have been sufficient.
Occasional packet loss within the NIC has historically been acceptable, particularly due to the prevalence of reliable protocols such as TCP.
Most high speed NICs are designed with only two physical network ports to satisfy redundancy and failover requirements.
With the advent of higher speed 10 Gb/s Ethernet (which is moving quickly to 40 Gb/s) and new standards relating to Ethernet flow control, the demands on a NIC have changed. Some of the recent extensions to Ethernet (e.g. 802.1Qbb and 802.1Qau) relate to protocols that will not tolerate packet loss, such as Fibre Channel over Ethernet (FCoE), Infiniband, and RDMA over Converged Enhanced Ethernet. Additionally, UDP (User Datagram Protocol) is becoming increasingly used in high performance environments in which extremely low latency data transfers are required, such as high frequency stock market trading. These Ethernet extensions increase the buffering requirements at the NIC and some extensions (e.g. 802.1Qbb) require that a number of flows over an Ethernet connection are individually flow controlled, which in practice requires that each flow is provided with dedicated buffering. For example, the 802.1 Qbb standard suggests that 18 KB should be set aside for each priority flow—a four-port NIC having six priority levels would therefore require 432 KB of additional buffer memory, in addition to the buffer memory used on the main data path of the NIC (typically 128 KB per port).
NICs are also becoming increasingly likely to include a greater number of physical ports, with four-port NICs becoming common. This is being driven by the increased role for NICs that have the ability to perform switching between virtualised interfaces or physical ports, possibly with packet modification on-the-fly. These changes are placing further demands on the buffering capabilities of high speed NICs.
There is therefore a need for a network interface device having a more efficient buffer memory architecture which provides improved buffer memory utilisation in real-world network conditions.